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ABSTRACT 

This booklet first looks at principles in evaluating 
services for people with mental retardation and strategies for 
evaluating and monitoring community services. It then describes two 
programs which involve parents/families in the evaluation and 
monitoring of community services in New Zealand. Basic principles 
include the importance of written standards, development of policy by 
service users, and linking of evaluation/monitoring systems with 
government standards. Several existing evaluation systems and 
accreditation procedures are briefly summarized. A two-level 
evaluation and monitoring system for community homes has been 
developed in New Zealand. At Level 1 local parents/consumers monitor 
homes and day programs on a frequent basis using a Key Points 
protocol which focuses on three areas: (1) the environment, (2) 
options and activities, and (3) management and staff, ^raining is 
provided for the parents, consumers, and volunteers no constitute 
these teams. At Level 2 less frequent but more comprehensive 
evaluations, also with heavy parent and consumer participation, 
assess quality of life issues by means of a 10-step process. The 
final section of the booklet lists lessons learned from 
implementation over the past 12 years including the low cost, less 
than 1 percent of total program costs on a nationwide level. An 
appendix lists other evaluation and monitoring methods. (Contains 14 
references.) (DB) 
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Terminology 

"Evaluation" and "Monitoring" are words that are often used 
interchangeably. Although we generally use the term "Evaluation and 
Monitoring" in all our work, strictly speaking we would use "evaluation" 
to describe a full length examination of a service designed not only to 
see that adequate standards exist but designed also to help new 
developments take place within a service. 

We use the term "monitoring" to describe the short examination of 
services made by teams on a regular basis. Such teams essentially focus 
on monitoring that adequate standards exist. If they feel the service needs 
a major revamp they will request a full scale evaluation. 
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Foreword by the President of the ILSMH 



In 1988 the League first published "Quality Evaluation Guidelines". 
I am pleased that we are now able to publish this practical description 
of the methods used in New Zealand to evaluate services using 
parents,consumers and staff working together. 

In 1991 as President of the League I was able to visit New Zealand 
and was encouraged to find that, at the opposite end of the earth, 
NZ supports community living services, and has developed a 
sophisticated system for evaluating and monitoring services. 

I believe we can all learn how to improve services by studying this 
booklet and thank my friend Angus Capie for sharing "the New 
Zealand Experience". 



Victor Wahlstrom 
Stockholm 1992 
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1. Developing an evaluation and monitoring 
programme for community services 



The second half of the Twentieth Century has seen a continuous and 
almost universal commitment to the provision of services for people 
with mental handicaps in community settings. 

Countries that for a hundred years had kept people in segregated 
institutions, hospitals or training centres, have almost all sought 
ways of integrating people with mental handicaps back into ordinary 
living situations. 

Countries that for various reasons have never possessed traditional 
mental handicap hospitals have begun to develop community 
services. 

This move into community living stemmed from a number of 
developments internationally, but primarily from 

a) the demands made by the international parents' movements that 
developed around 1950 (Beasley 1979) 

b) the acceptance of Normalization theory spelt out by 
Bank-Mikkelsen (1969) and Bengt Nirje (1969) in Scandanavia 
in the 1960's, and by Wolfensberger (1972) in North America a 
few years later. 

In North America the parallel emergence of the Civil Rights 
Movement undoubtedly assisted the cause of people with mental 
handicaps. 

Although most parents and families are very supportive of 
community living, it is widely recognized that simply living in a 
house in an ordinary street does not necessarily mean one becomes 
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part of the community. Many families and advocates are concerned 
that small community homes may become isolated and that there is 
an increased danger of poor standards developing in such situations. 

For over twenty years it has been recognized that as we develop 
small community services we should also develop ways of 
monitoring the services to ensure that the basic standards we seek 
are maintained. It is also recognized that services should be 
evaluated regularly to see how service providers can develop more 
responsive services. The challenge today is not simply to work out 
ways of monitoring and evaluating services - that is no great 
problem. The real challenge lies in being able to do three things: 

1) Develop strategies that enable services to be monitored regularly. 

2) Enable consumers and their families to determine what is 
appropriate in these services. 

3) Allow developmental evaluations to assist service providers to 
maintain quality at an affordable cost. 

In this document we will look at some of the principles involved in 
evaluating services, and examine some strategies developed to 
evaluate and monitor a range of community services. We shall also 
describe in some detail two programmes developed with 
parents/families that are being used to evaluate and monitor 
community services throughout New Zealand. 
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2. Principles involved in developing evaluation 
strategies 



Many different methods have been developed for evaluating and 
monitoring community services (a few examples are listed in 
Appendix I). 

Some programmes have been exceedingly expensive and complex. 
Others have consisted simply of short checklists given to 
parents/volunteers without at the same time, giving them any 
training or guidance in how the Checklists could best be used. 

Parents, family members and volunteers, like staff with professional 
training, should not be expected to participate in service evaluation 
without having undertaken an appropriate training programme. Such 
training is necessary to ensure consistency in approach, clarity in 
results, and to ensure that everyone understands that the task is one 
that is itself valued. 

BASIC PRINCIPLES 

1. The Society or Agency must have some written standards, or a 
philosophy and policy which tells members, service users, staff, and 
the funding agencies what is acceptable. It is not possible to evaluate 
a service without standards based on a clear philosophy. 

Examples of such statements are the Canadian Association for 
Community Living's "Community Living 2000" (1987), and IHC's 
New Zealand Philosophy and Policy (1990). 

Standards statements will vary from culture to culture, and will also 
be influenced by the economic state of a country. It is clearly 
inappropriate to have a standard for a community service that 
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demands material standards that are significantly better than those 
enjoyed by ordinary residents of that community. 

2. The philosophy/policy should be determined and developed by 
users of the service, their families and advocates. 

Most countries which developed segregated services did not do so 
at the request of families, but rather because of the advice of 
professionals, believing that "professionals know best" (See 
Wolfensberger, 1975). As a result people were removed from their 
family networks - often at a large emotional cost to the family, and 
at a large financial cost to the country. The failure of professionals 
to accept the desires of families/advocates for nonsegregated 
services is regrettable but common. Professionals must recognize 
the on-going interest and role of family members in service quality. 

3. Although the evaluation and monitoring system must reflect the 
preferences of service users (consumers), wherever possible it 
should be linked in with government standards. Where a 
government has not yet clearly spelt out 'Standards for Community 
Services', agencies should invest time in seeking government 
support for such Standards. Government support for community 
living programmes can help everyone. 

SOME SYSTEMS OF EVALUATION 

The development world wide of a range of community services owes 
a great deal to the many unheralded and unpublicized parents who 
dared to challenge "the Establishment" and "the Medical Model" by 
campaigning for community options. The progress made in 
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community living also owes a great deal to Wolf Wolfensberger and 
his associates (1973, 1975). 

Wolfensberger further developed Normalization theory, and later, 
argued that it was more useful to refer to Normalization as Social 
Role Valorisation (Wolfensberger, 1983). 

PASS and PASSING 

Wolfensberger and colleagues developed two methods for 
evaluating the quality of human services - PASS (Program Analysis 
of Service Systems) and PASSING (Program Analysis of Service 
Systems: Implementation of Normalization Goals). Both systems 
follow essentially the same procedure, although tUey use different 
measurements of service quality. 

PASSING (Wolfensberger and Thomas 1983) is an approach that 
all service providers and evaluators should examine carefully. 
PASSING states it has two major purposes: 

1 . To assess the normalization quality of any human service, 

2. To teach the principle of v Jrmalization." Wolfensberger and 
Thomas describe in detail Jie whole rationale for PASSING. It 
is interesting to note that PASSING does not attach any special 
virtue in using parents or family members. It emphasizes that 
Team Members must have undergone an approved PASSING 
training event. 
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In an effort to ensure that the principles are not watered down, 
materials used and strategies applied in PASSING are rigidly 
controlled. 

In New Zealand we found this limited our use of the material in 
evaluation and monitoring services. 

OTHER APPROACHES - ACCREDITATION PROCEDURES 

In a range of countries, governments have supported the 
establishment of Councils which are empowered to decide whether 
a service provider should continue to receive government funds. 

An example of this approach is seen in the "Standards for Services 
for People with Developmental Disabilities" (ACDD 1990). Such 
procedures are designed to ensure government funding is directed 
to services that meet stipulated conditions. 

This may ensure that at the time of accreditation the quality of service 
is high, but does not necessarily guarantee monitoring of service 
quality between accreditation visits. Family members or workers 
in the field will tell everyone of how quickly quality can change. 
Furthermore, the complexity of the Accreditation model, and the 
high costs involved, mean that in many places Accreditation is not 
a viable proposition. 

CONSUMER INVOLVEMENT 

Some accreditation procedures also insist that the team members or 
"surveyors" must be people with professional training in a discipline 
traditionally associated with this field. It could be argued that such 
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a policy devalues the role that family members can have in ensuring 
the needs of families are well met. Parents of people with a mental 
handicap often feel shut out by such procedures. Professionals 
come, professionals go, but families are there for ever, and although 
they acknowledge there is much they can learn from professionals, 
we believe that parents' knowledge base about their son/ 
daughter/sibling is of significant value and ultimately should be 
acknowledged in service planning. 

A further development of the policy of consumer involvement has 
been an increasing commitment to involving people with mental 
handicaps themselves in both developing policy and in being 
members of evaluation and monitoring teams. A separate 
monograph describing work in this area is in preparation. 
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3. A consumer controlled approach - the New 
Zealand experience 



New Zealand - a small South Pacific nation (population = 3 million) 
- established its major Parent organization (now known as IHC) for 
people with a mental handicap in 1949 . 

As successive governments failed to provide alternatives to 
institutions, IHC began to establish a range of educational, 
residential and day programmes. 

When the government accepted that the provision of education was 
a state obligation for all in 1989, IHC transferred all its educational 
services tc the State Education sector. 

Today IHC provides residential services for 3000 people and day 
programmes for over 4000 adults. Day programmes vary from 
sheltered workshops to supported employment programmes. 

Residential services range from supporting people living without 
live-in staff in apartments, to homes that have staff 24 hours a day 
7 days a week. Virtually all homes have 5 or fewer residents. 

In the late 1970's, influenced by the work of the Canadian National 
Institute for Mental Retardation (now the G. Allan Roeher Institute), 
a NZ group began to develop evaluation procedures for community 
homes (Standards for Residential Services, 1979). 

Many of the procedures and practices were a direct reflection of the 
work done in Canada. From the beginning of the NZ effort, 
however, parents/family members were seen as essential partners 
in the operation. 
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This programme for evaluation of residential services in IHC, 
although it went a long way in helping IHC develop its residential 
philosophies, had a number of deficiencies. By evaluating residential 
homes singly, it proved expensive and unable to regularly review 
all programmes. By focussing on residential services, it tended to 
ignore what happened to people away from their homes. 

A review of progress being made led to the development of a two 
tier evaluation and monitoring system. 

a) Level One (KEYPOINTS) enables more frequent monitoring 
programmes to be carried out 

b) Level Two (PREM) consists of less frequent but more 
comprehensive evaluations that assess the quality of life that 
people are receiving across services. 

KEY POINTS PROCEDURES 

It was agreed that parents, advocates, and consumers trained in 
monitoring services should take responsibility for ensuring that all 
local services were reviewed regularly, and thus see that these 
services met all basic standards. 

At a local level, all homes and day programmes would be reviewed 
by local parents/consumers using KEY POINTS procedures. No 
one could participate without training, and parents/staff accepted 
that people with mental handicaps would be key participants. 

This was reen as being a particularly appropriate area in which 
people with mental handicaps could begin to participate directly in 
monitoring services. 
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KEY POINTS MONITORING 



Monitoring local services to ensure that they are maintaining a 
satisfactory standard involves teams of usually 3 people (ideally a 
parent, a volunteer and a person with a mental handicap) in a short 
2-3 hour visit to the service. The team systematically focuses on 3 
major areas: 

a. The Environment - what does the service look and feel like? 
Does it reflect the cultural values of the residents? 

b. Options and Activities - Do people have a say in the things in 
life that are important to them? 

c. Management and Staff - Do the people who work in the service 
get the support and training they need? 

After the team's visit (and ideally every service should be visited at 
least every 6 months) a brief report commenting on services is 
produced. 

This may commend staff for some innovative actions, and will often 
suggest minor changes in the service. If the team is not satisfied 
with the service, it can ask that a full evaluation be undertaken. 

TRAINING TEAM MEMBERS 

Parents, consumers and volunteers who wish to take part in Key 
Points monitoring have to undergo training. A typical Introductory 
Training Seminar would be as follows: 
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5.45 PM Introductions 



6.00 PM An Introductory Session on Key Points Monitoring 

- values - an introduction 

- its purpose 

- confidentiality 

- observing 

- asking questions/listening 
7. 15 PM A light meal provided 

8.00 PM Visit the service in teams, with an experienced team 
leader, to practice using the Checklist and methods 

9.00 PM Reporting 

- Discuss findings with other team members 

- Agree on what should be in a report 

- Write a mock report 

9.30 PM Conclusion 

- Share ideas 

- Protocol and Procedures 

- Evaluation of session 

Initial Training Sessions may run from 10am to 5pm or over some 
evenings depending on what consumers want. 
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On successful completion of such a session, participants may then 
move into observer status on teams, and receive further training 
sessions, before becoming team members. Not all people who 
volunteer for training become service evaluators. Those who are 
selected after preliminary training receive on-going support. 

Keypoints has proved to be a cost effective way of monitoring 
services on a regular basis. 

THE PREM PROGRAMME 

IHC always recognized that although regular monitoring of quality 
was necessary, it was also desirable that a procedure for more 
in-depth evaluation of services should be developed. 

IHCs National Director (Munro 1987) asked that steps be taken to 
ensure: 

a) the evaluations were removed from the services division of IHC. 

b) they provide his staff with ways of developing and improving 
services. 

It was therefore decided to establish a new parent body, called the 
Standards and Monitoring Board, that was independent of all service 
providers, and give it responsibility for developing a system for 
evaluating services in depth. 

This was done through a staff/volunteer structure, known as 
Standards and Monitoring Services (SAMS). SAMS has a Director 
and a number of regional staff, as well as a number of regional 
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Evaluation and Monitoring Coordinators (usually parents of people 
with mental handicaps), who are trained and experienced in 
conducting evaluations. 

This programme now known as PREM (Procedure for Evaluation 
and Monitoring), uses teams of trained people led by a Coordinator, 
to evaluate a range of community services used by people with a 
mental handicap. After a 3 or 4 day study visit, teams report back 
with recommendations. 

A PREM EVALUATION 

The PREM programme is flexible and it fits in with varying demands 
from the community. Although firmly committed to seeing that the 
emphasis remains on a parent/staff partnership, we know that it 
enables a wide range of people to participate and that it can easily 
be adapted to encompass different cultural values in other societies. 

When PREM was first developed it tended to focus strongly on what 
was happening to all people in a particular service, be it home or 
day programme. Experience has shown us that it is more useful if 
we randomly select some service users, and then evaluate the total 
lifestyle they are able to enjoy, irrespective of their main base. 

The following outline illustrates a typical PREM evaluation. 

STEPS IN A PREM EVALUATION. 

1 . Co-ordinator is advised of agency/service to be evaluated. 
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2. Co-ordinator collects standard set of information about the 
service, including names of service users, family members and 
staff in the service. 

3 . Co-ordinator selects team members from trained pool available. 
Selection is influenced by the nature of services being evaluated 
and experiences of team members. 

4. Interview schedule for evaluation is arranged, and team members 
briefed. 

5. Team members visit referred service, and other services or 
establishments that are providing services for the selected group 
of consumers whose life styles are being evaluated. Such services 
may include Day Programme Centres, Tertiary Education 
Centres, Employers. 

6. Team interviews consumers, parents, family members and 
friends, plus staff of the agency. 

7. Team meets after visit and agrees on content of Draft report. 

8. Draft report sent to service provider. 

9. Meeting held with service provider and users to finalize report. 

10. Report finalized and schedule agreed for implementing the 
recommendations of the report. 
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SERVICE DEVELOPMENT 



The focus of PREM is on how we can best help service providers 
do better. 

Service providers therefore negotiate with the teams as to how 
services might be best developed. PREM teams do not have power 
to cut off funding - indeed they do not seek this power. However 
if any service fails to meet minimum standards then a team may 
recommend that the Government funding agency should consider 
such action. 

PREM teams emphasize that evaluation is about improving the 
quality of services. Parent/consumer controlled evaluation and 
monitoring aims to see that the quality of life offered to people with 
a mental handicap is maintained and improved. We believe this can 
only be achieved by co-operation with service providers. 
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4. The way ahead 



Collection of data about quality of services over a period of twelve 
years has taught us a number of things: 

1 . Evaluation and Monitoring of services need not be expensive. 
In NZ it costs less than 1% of the total cost of providing services 
to run a nation-wide evaluation programme. 

2. Evaluation and Monitoring is best developed and controlled by 
parents, consumers, families and staff working in partnership. 
Where evaluation is controlled only by professionals, it becomes 
so expensive that most places cannot afford it on a regular basis. 
A programme like PREM is a positive and valued means by 
which families can influence service provision. 

3. Evaluation and Monitoring will be more successful if it is done 
in partnership with government, eg. in New Zealand we have 
worked in partnership with the government to develop a 
Standards Document for Residential Services that reflects closely 
the basic philosophies on which PREM is based. 

4. Evaluation and Monitoring team members must be trained. 
Although we recognize the ongoing involvement of parents in 
the lives of their family member, we also recognize that everyone 
must be trained for the specific application of their own 
knowledge to an Evaluation and Monitoring situation. 

5. Although initially parent controlled organisations may be the 
ones most willing to look at Evaluation and Monitoring in this 
way, (as distinct from a process for meeting government 
standards), it is possible with a combination of persuasion and 



9 j Page 19 



encourage other providers - private or public - to believe that 
regular evaluation of sen/ices is not only good for the clients, 
but good for the providers. To this end it is important to foster 
a working partnership with service providers. 

6. It is necessary to ensure that resources are available for the the 
on-going support of parents and others. Practices and policies 
evolve, and everyone needs continuing opportunities to learn. 

7. Procedures such as PREM and KEYPOINTS are effective but 
can only remain so if all concerned make a real commitment. 
Boards of evaluation agencies must value and support 
parent/family involvement. 

8. Service providers are constantly being challenged by users to 
enhance and develop services. It is equally important that those 
involved in evaluation and monitoring are also prepared to 
continually update methods and systems. 
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Standards and Monitoring Services 
(SAMS) was established as a separate 
parent controlled body by IHC, New 
Zealand's major organisation for people 
with a mental handicap. 

A major objective of SAMS is to work 
with service providers to improve the 
quality of services offered to people with a 
mental handicap. 



26 




The International League of Societies for Persons with Mental Handicap 
(ILSMH) 



Founded in 1960 by representatives of To achieve these objectives, the League, 

societies of parents of mentally handi- with the help of its member societies, 

capped persons, friends and professionals organizes international symposia of ex- 

in the field of mental retardation, the perts, regional conferences and world con- 

ILSMH is devoted to defending the rights gresses. The League publishes the results 

and interests of mentally handicapped per- of these efforts, an international newslet- 

sons without regard to hationality , race or ter translated in four languages, and vari- 

religion. The League addresses problems ous pamphlets on topics of general interest, 

related to mental retardation: among them The League has consultative status with 

are prevention, diagnosis and early UNESCO, UNICEF, ILO, WHO, 

treatment, education and training, ECOSOC and the Council of Europe, and 

economic security, social welfare and has official relations with the European 

integration, guardianship, interfamilial re- Communities and other international or- 

lations, due process of law and public ganizations interested in handicapped 

education. persons. 



The main objectives of the League are: 

— to determ ine, with the help of persons 
with mental handicap, their families 
and specialists,what is required for 
these persons to live as close to nor- 
mal lives as possible; 

— to disseminate helpful information to 
and promote con tact between member 
societies; 

— to encourage the creation of new 
societies; 

— to initiate and develop contacts with 
international organisations, govern- 
mental andnon-governmental,inorder 
to speak on behalf of membersocieties; 

— to promulgate the basic principles set 
forth in the UN Declaration on the 
Rights of Mentally Retarded Persons. 
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